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NOTICES 
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(DA)  position  unless  so  designated  by  other  authorized  documents. 
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Reproduction  of  this  document,  in  whole  or  part,  is  prohibited  except  by  permission  of 
the  Director,  TRAC,  ATTN;  ATRC,  255  Sedgwick  Avenue,  Fort  Leavenworth,  Kansas 
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DISTRIBUTION  STATEMENT 
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SECTION  1.  PROBLEM  DEFINITION 


1.1,  BACKGROUND 

The  mounting  economic  challenges  faced  by  the  United  States  and  its  military  have 
forced  decision  makers  to  develop  a  way  in  which  to  holistically  examine  those  requirements  that 
drive  capability  development,  acquisition,  and  sustainment  in  order  to  determine  if  either  current 
or  proposed  programs  are  aligned  to  meet  critical  national  and  defense  strategies  as  well  as  Army 
plans  (U.S.  Army,  2012). 

Early  in  2010,  then  Secretary  of  Defense  Robert  M.  Gates  began  his  campaign  to  fix 
Department  of  Defense  (DoD)  spending  habits  through  acquisition  reform  and  overhead 
efficiency  savings.  His  goal  was  to  institute  a  department-wide  “culture  of  savings”  that  better 
aligned  expectations  with  results  and  resources.  This  vision  was  an  extension  of  Secretary  Gates’ 
program  reductions  in  April  of  2009,  which  eliminated  several  big-ticket,  underperforming 
weapons  projects.  Specifically,  the  Secretary  had  directed  each  of  the  services  to  review  its 
programs  and  administrative  structure  to  “find”  a  collective  $100  billion  in  cost  reductions  over 
the  next  five  years  (AUSA,  2010). 

Secretary  Gates  outlined  his  position  that  he  wanted  to  eliminate  the  extreme  peak/valley 
characteristics  of  defense  spending  that  are  so  disruptive  and  detrimental  to  both  the  planning 
and  execution  functions  of  programming.  While  valley-like  cuts  are  obviously  disruptive  to  long¬ 
term  programs,  high  peaks  are  equally  disruptive  in  that  they  can  cause  programmers  to  “throw 
money”  at  a  problem  rather  than  figure  out  ways  to  address  scheduling  issues,  technology 
maturation  problems,  or  general  design  faults.  In  exchange  for  a  focus  on  good  stewardship  of 
taxpayer  resources  and  overall  spending  reform.  Secretary  Gates  hoped  to  win  congressional 
support  for  stabilizing  the  defense  budget  with  a  sustainable,  inflation-relevant  growth  rate.  He 
indicated  that  his  target  was  a  1  percent  real  growth  rate  with  2-3  percent  internal  savings  for  3- 
4  percent  total  actionable  growth  in  the  next  five  years.  The  Army  Capability  Portfolio  Review 
(CPR)  process  has  been  at  the  leading  edge  of  this  effort  to  increase  efficiencies  (AUSA,  2010). 

On  February  22,  2010,  the  Secretary  of  the  Army  (SA)  directed  the  Under  Secretary  of 
the  Army  (USA)  and  the  Vice  Chief  of  Staff  of  the  Army  (VCSA)  to  implement  a  CPR  pilot 
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process  to  conduct  an  Army-wide,  all  eomponents  revalidation  of  the  operational  value  of  Army 
requirements  within  and  aeross  eapability  portfolios  (Keenan,  2013).  The  CPR  then  is  designed 
to  support  the  Army’s  overarehing  and  strategie  level  goals  of  fielding  an  effective,  flexible, 
affordable,  and  modem  foree.  The  SA  then  implemented  a  CPR  proeess  with  three  distinct 
objectives:  to  revalidate  Army-wide  system  requirements  aeross  all  eomponents;  to  align 
resourees  with  warfighting  and  Soldier  priorities;  and  to  institutionalize  resource  and  aequisitions 
along  eapabilities-based  planning  proeesses  to  provide  flexibility  for  the  future  (AUSA,  2010). 

“The  starting  point  is  determining  what  the  requirement  is.  What  doeuments  the 
requirement?  Is  it  a  valid  requirement?”  said  Theresa  M.  Sherman,  Division  Chief,  Capability 
Portfolio  Review  and  Integration,  G-3/5/7.  “And  then  you  look  aeross  the  DOTMLPF  (Doetrine, 
Organization,  Training,  Materiel,  Leadership  and  Edueation,  Personnel,  and  Faeilities)  and  see, 
have  we  aceepted  too  mueh  risk  or  ean  we  take  more  risk;  are  we  still  in  syne  with  the  need,  with 
the  threat?  Do  we  have  any  gaps  or  redundancies?  . . .  Where  do  we  antieipate  the  threat  will  be, 
and  where  do  we  want  to  go  over  time?  What’s  the  strategy?”  Answers  to  these  questions  depend 
in  large  part  on  information  provided  by  the  U.S.  Army  Training  and  Doetrine  Command 
(TRADOC)  and  Program  Executive  Offices  (PEOs)  (Roth  &  McCouch  III,  2012). 

As  explieit  dollar  savings  are  directed  from  DoD,  the  other  serviees  are  following  the 
Army’s  lead  in  reorganizing  their  aequisition  proeesses  and  meehanisms  to  generate  more 
affordability,  oversight,  and  effieieney  ineentives  aeross  the  full  range  of  portfolios.  In  all 
likelihood,  the  CPR  proeess  is  a  permanent  addition  to  both  the  Army’s  and  DoD’s  aequisition 
models,  sinee  the  affordability  initiatives  from  the  Seeretary  of  Defense  are  meant  to  eontinue 
through  the  next  5  years.  The  CPR  proeess  will  eontinue  to  grow  and  evolve  to  ineorporate  more 
areas  and  apply  more  expertise  and  analysis  in  order  to  provide  the  warfighters  what  they  need, 
when  they  need  it,  and  at  the  “best”  priee.  It  is  designed  to  maximize  fiseal  resouree  expenditures 
as  well  as  ensuring  that  Soldiers  reeeive  the  very  best  eapability  at  the  best  value  to  the  taxpayer 
(AUSA,  2010). 

1.2,  PROJECT  PURPOSE 

To  provide  a  literature  review  and  doeument  historieal  Capability  Portfolio  Reviews 
(CPRs)  while  using  the  work  done  by  the  Training  and  Doetrine  Command  (TRADOC)  Analysis 
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Center  -  White  Sands  Missile  Range  (TRAC-WSMR)  on  the  IBCT  CPR  as  a  case  study  for 
exploring  alternative  trades  analysis  methodologies,  to  include  the  use  of  Bayesian  statistical 
methods,  in  order  to  better  inform/influence  decision  makers. 

1.3.  CONSTRAINTS,  LIMITATIONS,  &  ASSUMPTIONS 

Constraints  limit  the  study  team’s  options  to  conduct  the  study. 

•  The  project  will  be  completed  by  30  April  2014. 

Limitations  are  a  study  team’s  inability  to  investigate  issues  within  the  sponsor’s  bounds. 

•  Survey  of  TRAC  support  to  a  formational  CPR  case  study  will  be  limited  to  the 
IBCT  as  TRAC  did  not  participate  in  the  Rifle  SQD  CPR  process. 

•  Lack  of  sufficient  CPR  documentation  precludes  an  in-depth  survey  of  historical 
functional  CPR  analysis  techniques. 

Assumptions  are  study  specific  statements  that  are  taken  as  true  in  the  absence  of  facts. 

•  CPR  process  input  will  be  provided  by  the  G-3/5/7  DAMO-CIP. 

•  Alternate  statistical  /  Bayesian  techniques  applied  to  the  IBCT  CPR  will  be 
applicable  to  all  types  of  CPR’s. 

1.4.  METHODOLOGY 

While  a  description  of  the  CPR  process  is  certainly  in  order,  and  will  be  given  later  on  in 
this  technical  report,  this  effort  will  focus  less  on  the  CPR  process  as  a  whole  and  more  on  an 
alternate  trade  space  analysis  methodology  as  well  as  a  way  to  present  decision  makers  with  the 
correct  information  in  order  to  make  the  most  informed  decisions  with  respect  to  future  Army 
combat  systems  and  formations. 

This  project  will  consist  of  three  related  but  parallel  efforts  (see  Figure  1).  The  first  will 
focus  on  an  in  depth  literature  review  of  the  CPR  process  as  well  as  a  summary  of  historical 
CPRs  in  order  to  give  this  effort  the  proper  context,  followed  sequentially  by  documenting  the 
best  and  worst  decision  analysis  practices  using  those  historical  CPRs  as  use  cases.  At  the  same 
time,  the  project  team  will  be  reviewing  both  the  IBCT  CPR,  as  well  as  surveying  alternate 
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decision  analysis  methodologies,  to  include  Bayesian  statistical  methods.  Using  the  goal 
programming  work  done  by  TRAC-WSMR  on  the  IBCT  CPR  as  a  case  study,  we  will  then 
explore  what  alternate  method(s)  is  (are)  most  appropriate,  what  additional  data  requirements 
there  might  be,  and  then  apply  said  method  in  a  small  use  case.  We  hope  to  find  a  better  way  in 
which  we  can  both  execute  analysis  and  present  information  to  decision  makers.  Lastly,  the 
project  team  will  document  their  efforts,  as  well  as  provide  support  for  TRAC-WSMR’ s  effort  in 
working  on  the  ABCT  CPR,  as  appropriate. 


Project  Methodology 
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Figure  1:  CPR  Analysis  Methodology 
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SECTION  2. 


THE  CAPABILITY  PORTFOLIO  REVIEW 


2.1,  PURPOSE 

The  overarching  CPR  process  is  designed  to  synchronize  the  planning, 
programming  and  budgeting  with  feedback  from  combatant  commanders  and  lessons  learned 
from  Iraq  and  Afghanistan,  all  while  leveraging  emerging  technologies  with  affordability.  The 
CPR  process  is  a  critical  element  in  restoring  the  balance  to  the  Army  and  the  equipping/fielding 
strategies  (AUSA,  2010). The  intent  or  purpose  of  the  CPR  process  is  to  conduct  an  Army-wide 
(all  components)  revalidation  of  requirements  while  holistically  examining,  validating, 
modifying,  or  making  termination  recommendations  for  the  requirements  that  drive  capability 
development,  acquisition,  and  sustainment.  The  process  operates  with  four  central  goals  in  mind; 

1 .  Establish  the  ability  to  examine  and  modify  investment  portfolios. 

2.  Develop  an  understanding  of  requirements  driving  investment,  procurement,  and 
sustainment. 

3.  Reconcile  requirements  across  portfolios. 

4.  Validate,  modify,  and  terminate  investment  and  /  or  procurement  strategy  upon  the 
reconciliation  of  requirements,  with  the  goal  of  the  reconciliation  piece  to  ensure  that  funds  are 
programmed,  budgeted,  and  executed  against  validated  requirements,  cost,  and  risk-informed 
alternatives  (AUSA,  2010). 

2.2.  DEFINITIONS  &  TYPES  OF  CPRS 

Before  we  begin  talking  about  the  CPR  process  as  a  whole,  let  us  first  define  a 

few  key  terms: 

Capability;  The  ability  to  achieve  a  desired  effect  under  specified  standards  and 
conditions  through  a  combination  of  means  and  ways  across  DOTMLPF  in  order  to  perform  a  set 
of  tasks  to  execute  a  specified  course  of  action  (Department  of  Defense,  2008). 

In  order  to  identify  a  capability  then,  we  must  determine; 

1 .  What  we  need  to  accomplish,  or  the  desired  effect. 
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2.  Why  we  need  to  aecomplish  it,  or  the  driver. 

3.  Who,  where,  and  when,  or  the  requirements  /  standards  and  eonditions. 

4.  How  we  will  aeeomplish  it,  or  the  ways  and  means,  usually  through  a  DOTMLPF 
assessment. 

Portfolio:  A  set  of  existing  programs  eurrently  funded,  plus  any  additional  programs  of 
initiatives,  that  provide  eapabilities  within  the  domain  under  eonsideration. 

Capability  Portfolio:  A  eolleetion  of  grouped  eapabilities  as  defined  by  joint  eapability 
areas  and  the  assoeiated  DOTMLPF  programs,  initiatives,  and  aetivities. 

Trade  Space:  The  set  of  system  parameters,  attributes,  and  eharaeteristies  required  to 
satisfy  performanee  standards. 

Trade  Space  Analysis:  A  process  that  identifies  potential  opportunities  to  allocate 
resources  in  order  to  optimize  the  balance  between  operational  capability  and  financial 
efficiency. 

There  are  3  types  of  CPRs,  a  functional,  formational,  and  cross-portfolio.  The  functional 
CPR  follows  the  original  guidance  provided  by  GEN  Chiarelli  and  focuses  on  prioritizing 
capabilities  that  are  grouped  together  in  terms  of  functionality.  For  example.  Aviation,  Mission 
Command,  Combat  Vehicles,  Soldier  Systems,  and  Mobility/Counter-mobility,  are  instances  of 
functional  CPRs  and  ones  that  have  been  completed  by  the  Army  in  the  2011-2012  timeframe. 

A  formational  CPR,  on  the  other  hand,  focuses  on  the  type  of  Army  formation,  rather 
than  on  functionality,  and  provide  additional  formation  perspective  by  integrating  across 
functional  portfolios,  war  fighting  functions,  DOTMLPF  analysis,  and  Program  Evaluation 
Groups  (PEGs).  To  date,  there  have  only  been  two  formational  CPRs  completed.  Rifle  Squad, 
and  the  IBCT.  The  ABCT  CPR  is  nearing  completion  and  should  be  completed  by  late  summer 
of2014. 

The  final  type  of  CPR  is  the  cross-portfolio  CPR.  In  theory,  a  cross-portfolio  CPR  is  an 
aggregation,  at  the  Army  level,  of  formational  and  functional  CPRs  designed  to  inform  the  entire 
Army’s  portfolio  and  evaluate  it  against  DoD  and  Army  strategic  guidance  and  operational  plans. 
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As  of  completion  of  this  document,  none  of  this  type  has  been  attempted,  much  less  completed, 
though  the  Center  for  Army  Analysis  (CAA)  has  been  tasked  to  look  into  its  feasibility. 


2,3.  PROCESS 

Although  each  CPR  is  conducted  differently,  they  all  have  their  beginnings  with  the  USA 
and  VCSA.  Running  the  CPR  is  the  G-3/5/7,  as  the  staff  proponent  for  organizing  and  executing 
it.  Contributing  information  on  the  portfolios  are  representatives  from  across  the  programmatic 
spectrum,  including  the  TRADOC;  the  Program  Executive  Offices  (PEOs);  the  Deputy  Chief  of 
Staff,  G-8;  and  the  testing  community  (Roth,  2012). 

The  G-3/5/7  runs  the  meetings,  sets  the  schedule,  gets  the  subject  matter  experts  together, 
and  identifies  the  major  issues.  After  a  series  of  meetings  at  the  0-6  level,  there  are  a  few  at  the 
one  and  two  star  level,  followed  by  (if  necessary)  a  three-star  meeting  with  the  VCSA.  At  that 
point,  most  of  the  work  is  finished,  namely  a  detailed  review  of  the  portfolio  with  particular 
attention  to  selected  issues  (Roth,  2012). 

“We  try  to  get  from  [TRADOC]  what  are  the  requirements,  and  how  old;  revalidate  them; 
and  try  to  find  some  form  of  strategic  context  so  you  could  look  at  importance  and  redundancy  to 
help  stack  them  and  there  would  be  some  form  of  strategy  upfront,”  said  Dr.  David  Markowitz, 
Technical  Advisor  to  the  Deputy  Chief  of  Staff,  G-3/5/7.  Eor  instance,  for  a  missile  defense 
portfolio,  the  initial  focus  would  be  on  the  range  of  threats,  then  on  the  needs  of  the  force, 
resourcing,  and  how  much  the  systems  cost.  This  information  then  helps  in  determining  how 
many  of  each  system  the  Army  needs  and  how  quickly  the  Army  needs  it,  he  said  (Roth,  2012). 

Einally,  the  DOTMLPE  is  applied  to  each  of  the  systems,  and  is  then  refined  in  a  series  of 
meetings  at  successively  higher  levels  until  it  gets  to  the  three-star  level,  the  Deputy  Chief  of 
Staff,  G-3.  “And  then  you  see  that  final  body  of  work  and  you  say  good,  the  doctrine  is 
validated,”  Markowitz  said  (Roth,  2012). 

The  G-3  then  presents  the  information  to  the  VCSA,  who  in  the  end  gets  a 
recommendation  or  status  of  the  portfolio  and  its  health,  and  is  asked  to  validate  and  approve  of 
what  was  done  in  terms  of  prioritization  and  overall  portfolio  strategy  (Roth,  2012). 
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The  VCSA,  in  turn,  presents  the  recommendation  to  the  USA,  sometimes  after  several 
sessions  with  the  staff.  Generally,  when  the  USA  approves  a  recommendation,  it  will  go  to  the 
CSA  for  approval.  Ultimately  there  may  be  a  need  to  modify  the  requirement,  initiate  additional 
review,  or  pursue  DoD  support  to  change  the  program  (Roth,  2012). 

The  G-8  provides  information  on  costing  and  affordability  to  support  the  CPRs,  but  that 
often  comes  directly  from  the  PEOs.  The  CPR  process  is  meant  to  be  very  open,  and  is  centered 
around  what  the  best  way  is  to  provide  the  most  effective,  capable,  and  cost-effective  Army. 

While  the  intent  or  purpose  of  the  CPR  process  is  clear,  the  execution  is  not.  Currently 
there  is  no  standardized,  documented  process  for  conducting  a  CPR,  and  there  is  no  Army 
Regulation  that  specifies  the  ways  in  which  a  CPR  should  be  accomplished.  There  was  an  effort 
by  TRADOC  and  ARCIC  in  late  2013  in  an  attempt  to  codify  the  process.  HQ  TRADOC 
published  a  draft  task  order  in  December  2013  laying  out  the  CY  2014  and  2015  CPR  schedule 
for  both  formational  and  functional  based  CPRs,  as  well  as  providing  some  guidance  on  analysis 
and  briefing  content.  A  final,  non-draft  task  order  has  yet  to  be  published  (as  of  publication  of 
this  report).  ARCIC  published  a  CPR  SOP  in  early  2014,  describing  the  purpose,  goals, 
objectives,  and  components  of  a  formational  CPR,  as  well  as  portfolio  assessment  and  strategy. 
As  with  the  TRADOC  draft  task  order,  no  final  version  of  this  SOP  has  been  published. 
Additionally,  in  a  report  submitted  to  the  Secretary  of  the  Army  in  January  of  201 1  entitled, 
“Army  Strong:  Equipped,  Trained  and  Ready,  Einal  Report  of  the  2010  Army  Acquisition 
Review”,  the  panel  recommended,  among  other  things,  that: 

Capability  Portfolio  Reviews  (CPRs)  are  intended  to  conduct  an  Army-wide,  all¬ 
components  revalidation  of  requirements.  The  approach  is  to  holistically  examine, 
validate,  modify  or  make  recommendations  to  terminate  requirements  driving  capability 
development,  acquisition  and  sustainment  across  a  series  of  portfolios.  Having  the  VCSA 
and  ASA(ALT)  co-chair  the  first  session  of  the  materiel  CPRs  would  further  restore  the 
traditional  partnership  discussed  above.  Codifying  CPRs  in  an  Army  Regulation  (AR) 
will  give  assurance  that  the  process  will  he  continued  when  leadership  changes.  The 
responsibilities  and  accountability  of  participants  in  a  CPR  should  be  clearly  defined.  The 
CPRs  should  be  expanded  in  the  future  to  review  the  interdependencies  across  portfolios 
(McHugh,  2011). 

There  is,  however,  a  description  of  the  process  given  by  the  Army  Eorce  Management 
School  (AFMS)  in  their  “Capabilities  Development  and  System  Acquisition  Management 

Executive  Primer  (2013)”,  though  this  is  still  a  long  way  from  inclusion  in  an  AR. 
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The  process  has  two  phases,  each  chaired  by  different  individuals,  and  each  with  a 
distinct  and  separate  set  of  outputs  that  ultimately  inform  the  Army  FY  Program  Objective 
Memorandum  (POM).  HQDA,  DCS  G-3/5/7  is  the  lead  agency  for  CPR  coordination  and 
synchronization.  Each  of  the  phases  is  described  below: 

(1)  Phase  1:  Chaired  by  the  VCSA.  The  purpose  is  revalidation  of  the  operational 
value  of  Army  requirements  to  include  cost,  schedule,  performance,  life-cycle  sustainability  and 
the  Army’s  plan  to  manage  the  totality  of  the  requirement.  The  output  is  actionable 
recommendations  that  can  be  addressed  by  Army  senior  leadership  during  phase  2. 

(2)  Phase  2:  The  USA,  as  the  Army  Chief  Management  Officer,  chairs  this  phase. 
The  purpose  is  to  address  follow-on  analysis  from  phase  1  and  the  programmatics  (cost, 
schedule,  performance,  life-cycle  sustainment)  implications  of  the  recommendations  presented. 
The  product  is  actionable  recommendations  to  the  SA  to  validate,  modify,  or  terminate  research 
and  development  (R&D)  investment,  procurement,  and/or  life-cycle  sustainment  requirements 
within  capability  portfolio  accounts  for  the  current  POM,  in  development,  based  on  the  results  of 
the  CPRs. 

The  analysis  that  has  resulted  from  the  CPRs  conducted  under  the  program  has  clearly 
highlighted  the  utility  of  this  process  in  building  an  effective  and  affordable  modernization 
strategy.  The  resulting  recommendations  will  continue  to  assist  the  SA  in  establishing  future 
priorities  for  investment,  research,  development,  acquisition,  and  life-cycle  sustainment.  The  SA 
will  continue  to  rely  on  this  process  to  help  him  make  informed  decisions  on  behalf  of  the  Army. 

CPR’s  operate  concurrently  with,  but  do  not  supplant  the  authority  of  the  Army 
Requirements  Oversight  Council  (AROC),  Army  System  Acquisition  Review  Council 
(ASARC),  or  Configuration  Steering  Board  (CSB)  forums,  previously  discussed  (Keenan,  2013). 

2,4,  REVIEW  OF  CPR  LITERATURE 

This  section  is  primarily  focused  on  providing  a  brief  survey  of  the  portfolio  analysis 
tools,  theory,  and  trade  space  analysis  methodologies  developed  both  in  academia  or  those  used 
in  the  business  world  with  potential  applications  to  the  Army. 
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2,4,1.  Academia 


The  majority  if  the  CPR  literature  found  in  aeademia  has  both  its  roots  and  applieations 
in  the  finanee  world.  This  seetion  will  provide  a  short  survey  of  some  of  the  more  well-known 
portfolio  theory  ideas  and  eoneepts  that  deal  mainly  with  the  notions  of  risk  and  return. 

2.4. 1.1.  The  Basics  of  Modern  Portfolio  Theory 

Mueh  of  modern  portfolio  management  has  been  motivated  by  the  influential 
work  of  Harry  Markowitz  (Markowitz,  1952)  and  his  well-known  optimization  approach. 
Markowitz  demonstrated  how  stock  investors  could  select  an  efficient  set  of  portfolios  that 
would  minimize  the  standard  deviation  (risk),  subject  to  a  particular  portfolio  return  (expected 
return).  Markowitz  (1956)  showed  through  an  optimization  technique  that  investors  could 
virtually  eliminate  their  exposure  to  the  unsystematic  risk  associated  with  individual  securities, 
and  is  applied  in  the  context  of  a  fixed  investment  amount  for  the  portfolio.  The  unsystematic 
risks  are  those  risks  specific  to  the  business  or  industry.  This  ability  to  diversify  away  the 
unsystematic  risk  leaves  the  stock  investor  with  a  portfolio  containing  only  the  market-specific 
risks,  such  as  inflation,  purchasing  power,  etc...  Markowitz  demonstrated  that  with  only  a  limited 
number  of  properly  selected  stocks,  the  investor  could  virtually  eliminate  all  the  unsystematic 
risk  associated  with  individual  stocks,  leaving  only  the  generally  market  risk  (Walls,  2004). 

The  basic  assumption  of  modem  portfolio  theory  is  that  decisions  are  made  on  the 
basis  of  a  tradeoff  between  risk  and  return.  Return  is  measured  by  the  expected  value  or  mean  of 
the  probability  distribution  of  payoffs  for  the  stock  under  consideration.  Risk  is  then  measured 
by  the  variance  or  standard  deviation  associated  with  that  payoff  distribution.  In  addition,  one 
can  make  the  reasonable  assumption  that  investors  and  decision  makers  prefer  less  risk  to  more 
risk,  all  other  things  being  equal.  Or,  given  a  certain  expected  return  rational  investors  will 
always  prefer  portfolios  that  have  lower  risk.  Similarly,  given  a  certain  level  of  risk,  those  same 
investors  will  always  prefer  portfolios  with  higher  expected  returns  (Walls,  2004). 

In  some  portfolio  optimization  approaches,  risk  is  defined  as  the  standard 
deviation  of  returns  (i.e.,  net  present  value)  of  the  portfolio  of  assets.  Indeed,  standard  deviation 
is  a  commonly  used  measure  of  risk  in  the  financial  markets  where  return  distributions  are 
generally  normally  distributed.  Though  standard  deviation  is  utilized  as  a  measure  of  risk  in  the 
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finance  literature,  this  measure  is  better  defined  as  a  statistieal  measure  of  uncertainty.  It  is,  in 
effeet,  a  measure  of  dispersion  around  the  mean  value.  Unlike  the  deeision  maker  who  may 
characterize  risk  as  that  portion  of  the  uncertainty  that  has  “downside”,  the  standard  deviation 
measure  does  not  differentiate  between  “downside”  and  “upside”  uncertainty.  In  this  context, 
portfolio  analysis  based  on  standard  deviation  (as  deseribed  by  Markowitz)  eonsiders  extremely 
high  and  low  returns  equally  undesirable  (Walls,  2004). 

It  is  important  to  note  that  in  some  cases,  returns  on  investments  may  not  be 
normally  distributed.  In  many  cases  the  distribution  of  outcomes  may  have  skewed  value 
distributions  with  high  probability  of  achieving  low-value  outcomes  and  small  probability  of 
achieving  high-value  outcomes.  In  certain  cases,  it  may  be  more  appropriate  to  utilize  an 
alternative  measure  of  risk  such  as  semi-standard  deviation.  The  semi-standard  deviation 
measure  concentrates  on  redueing  losses  where  the  loss  point  in  the  semi-standard  deviation 
measure  is  defined  by  the  decision  maker.  However,  that  there  are  practical  complications 
associated  with  the  application  of  the  semi-standard  deviation  measure  in  portfolio  optimization. 
For  example,  there  are  issues  assoeiated  with  seleeting  an  appropriate  “loss  point”  for  a 
distribution  of  outcomes.  Also,  in  an  analysis  based  on  standard  deviation,  only  means,  variances 
and  covarianees  must  be  supplied  as  inputs  to  the  analysis.  In  a  semi-standard  deviation  analysis 
the  entire  joint  distribution  of  outcomes  is  required  in  order  to  perform  the  analysis  (Walls, 

2004). 


2.4.I.2.  Risk  Informed  Trade  Space  Analysis 

The  RAND  Corporation  developed  a  tool  for  trade  space  analysis,  and  published 
it  (only  in  draft  version)  in  a  paper  titled  “Developing  a  Methodology  for  Risk-Informed  Trade 
Space  Analysis  in  Aequisition”.  However,  at  the  time  of  the  writing  of  this  doeument,  the  report 
is  proprietary  and  not  eleared  for  open  publieation,  cireulation,  or  quotation.  We  mention  it  here 
only  for  completeness  sake  and  the  hope  that  it  will  be  eleared  for  publieation  in  the  near  future, 
as  it  appears  to  have  great  application  to  the  topie  of  this  effort. 

2,4,2.  Business 

Capability  Portfolio  Reviews,  if  we  can  generalize,  are  not  unique  within  either  the 

Department  of  the  Army,  or  the  Department  of  Defense.  Businesses  with  diverse  interests  also 
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conduct  such  reviews  in  an  effort  to  streamline  their  eompanies.  This  seetion  will  highlight  a  few 
of  the  tools  and  methodologies  used  in  industry  for  the  purpose  of  reviewing  a  portfolio. 

2. 4. 2, 1.  RAND ’s  Portfolio  Management  (PortMan)  Method 

Though  originally  designed  to  be  applied  to  Naval  Researeh  and  Development, 
this  partieular  portfolio  management  teehnique  developed  by  RAND  is  equally  applieable  to 
Army  investment  strategies.  RAND’s  PortMan  Researeh  and  Development  (R&D)  deeision 
framework  eomputes  the  Expeeted  Value  (EV)  of  an  R&D  projeet  as  the  produet  of  three  faetors: 
military  value  of  the  eapability  sought,  the  extent  to  whieh  the  performanee  potential  matehes  the 
level  required  to  aehieve  the  eapability,  and  the  projeet’s  transition  probability.  A  useful 
interpretation  of  this  equation  is  that  the  performanee  potential  seales  the  eapability  value  that  is 
based  on  aehieving  a  required  level  of  performanee,  whieh  is  then  further  redueed  by  the 
transition  probability  in  order  to  obtain  the  expeeted  value  of  the  researeh  (Silberglitt  et  ah, 

2004). 

The  purpose  of  PortMan  is  to  evaluate  a  defined  group  of  aetual  or  proposed 
projeets  and  to  provide  a  means  for  ereating  a  portfolio  from  them  that  maximizes  the  value  of 
R&D  investments.  It  does  not  generate  an  absolute  seore  for  the  total  portfolio  that  eould  be  used 
to  eompare  to  portfolios  of  other  projeets  or  to  proportionally  alloeate  funds  between  portfolios 
of  different  projeets.  PortMan  does  not  rely  on  the  EV  as  a  point  solution,  but  rather  ineludes  an 
estimate  of  uneertainty,  evaluating  and  eomparing  R&D  projeets  based  upon  all  three 
eomponents  of  the  EV,  ineluding  uneertainty,  plus  their  estimated  direetion  of  ehange  over  time. 
This  requires  evaluation  based  on  best  eurrent  information,  and  traeking  over  time  as  eonditions 
ehange  and  the  teehnology  matures  (Silberglitt  et  ah,  2004). 

An  important  feature  of  this  approaeh  is  that  it  allows  identifieation  of  those  R&D 
projeets  seeking  high-value  eapabilities,  but  may  have  diffieult  teehnieal  or  fielding  problems 
remaining  to  be  solved,  and  for  whieh  management  attention  may  have  the  greatest  leverage 
(Silberglitt  et  ah,  2004). 

This  framework  ean  be  used  with  a  group  of  subjeet  matter  experts  determining 
the  values  of  eapability,  performanee  potential,  and  transition  probability.  It  is  eapable  of 
eomparing  and  eontrasting  individual  and  groups  of  researeh  projeets  as  a  funetion  of  key 
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management  variables  sueh  as  their  potential  value  and  status  and  quality  of  transition  planning. 
It  also  provides  a  straightforward  and  logical  set  of  repeatable  steps  in  order  to  determine  the 
expected  value  of  research  projects,  together  with  a  measure  of  uncertainty  (Silberglitt  et  ah, 
2004). 


2.4.2.2.  RAND^s  PortMan  with  a  Linear  Program 

In  this  section,  we  describe  a  variant  of  PortMan  that  uses  a  linear  programming 
model  to  select  a  portfolio  that  consists  of  Army  Technology  Objectives  (ATOs),  the  highest 
priority  Army  S&T  projects.  This  portfolio  satisfies  the  Army’s  Force  Operating  Capability 
(FOC)  requirements  designated  for  this  group  of  ATOs  to  meet,  while  yielding  the  lowest  total 
remaining  lifecycle  cost  for  all  the  systems  developed  from  the  selected  ATOs.  The  model 
includes  two  classes  of  constraints.  First,  all  individual  FOC  requirements  must  be  fully  met. 
Second,  the  total  remaining  S&T  budget  for  the  selected  ATOs  must  not  exceed  a  given 
budgeted  amount.  The  linear  programming  model  used  here  requires  two  inputs:  (1)  the  EV  of 
the  ATOs  and  (2)  the  remaining  lifecycle  cost  of  the  systems  that  will  be  developed  from  the 
ATOs  (For  specifics  on  the  linear  program  model  formulation,  see  Chow  et  ah,  2009)  (Chow, 
Silberglitt,  &  Hiromoto,  2009). 

RAND  has  developed  an  approach  called  “gap  space  coverage”  to  make  EV 
estimates  of  how  well  each  ATO  could  meet  EOC  sub-requirements  defined  by  TRADOC.  This 
approach  is  based  on  the  multiplication  of  three  factors:  how  many  situations  encountered  by 
warfighters  to  which  the  system  derived  from  the  ATO  can  make  a  contribution,  the  size  of  the 
gap  space  in  an  EOC  that  the  system  can  mitigate,  and  the  size  of  the  contribution  that  the  system 
can  make  to  fill  the  gaps.  The  cost  has  three  components:  (1)  acquisition,  (2)  upgrade,  and  (3) 
operating  and  maintenance  (Chow  et  ah,  2009). 

Here  is  a  listing  of  some  of  the  applications  of  this  method: 

•  Determination  of  the  extent  to  which  the  EOC  requirements  would  be  met  if  all 
existing  ATOs  were  completed  and  their  systems  fielded,  thereby  allowing  the  Army 
to  trace  the  impact  of  ATOs  on  EOCs. 
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•  Identification  and  introduction  of  new  ATOs  for  which  existing  ATOs  leave  gaps  in 
meeting  FOC  requirements. 

•  Determination  of  the  subset  of  existing  ATOs  that  can  meet  all  individual  FOC 
requirements  at  the  lowest  total  remaining  lifecycle  cost. 

•  Determination  of  the  extent  to  which  each  of  the  individual  FOC  requirements  is 
exceeded. 

•  Determination  of  the  optimal  distribution  of  funds  across  FOC  requirements. 

•  Determination  of  which  set  of  ATOs  should  be  ranked  high  and  should  protected 
from  Army  budgetary  cuts.  (Chow  et  al,  2009). 

2.4. 2. 3.  RAND  Portfolio  Analysis  Tool  (PAT) 

PAT  is  not  a  model  in  the  usual  sense;  rather,  it  is  a  cross-platform  spreadsheet 
tool,  built  in  Microsoft  Excel,  which  facilitates  planning  by  presenting  information  in  a  way  that 
is  useful  to  senior  leaders.  PAT  can  use  a  variety  of  separate  or  embedded  models  as  sources  of 
input  data.  PAT  has  many  useful  features  (Davis  &  Dreyer,  2009): 

•  PAT  generates  stoplight  charts,  simple  color-scorecard  summaries  of  how  options 
rate  on  a  number  of  juxtaposed  criteria,  such  as  measures  of  capabilities,  risks,  upside 
potential,  and  costs. 

•  PAT  generates  its  summaries  from  more  detailed  considerations,  which  can  be 
viewed  by  drilling  down  to  a  level  that  provides  assumptions,  a  terse  logic,  and  a 
measure  of  rigor,  even  for  qualitative  assessments.  Two  levels  of  drilldown  are 
available. 

•  PAT  allows  the  analyst  to  quickly  recognize  key  assumptions  and  to  change  them 
interactively.  This  may  be  done  parameter-by-parameter  or  more  broadly.  These 
analyses  are  greatly  facilitated  by  the  MRM  feature. 

•  PAT  allows  the  analyst  to  quickly  change  how  summary  depictions  are  generated. 
Choices  include;  simple  linear  weighted  sums,  some  nonlinear  “weakest  link” 
methods,  linear  weighted  sums  with  threshold  constraints,  and  rank  ordering. 
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•  PAT  links  to  even  more  detailed  information,  sueli  as  that  of  an  embedded  or 
eonneeted  eapabilities  model,  data  generated  separately  from  a  eapabilities  model, 
empirieal  data,  or  struetured  judgments. 

•  Although  PAT  emphasizes  multi-objeetive  seoreeards,  it  also  generates  seores  of 
overall  effeetiveness  or  eost-effeetiveness  whieh  ean  be  used  for  marginal  or  ehunky 
marginal  analysis  about  how  to  spend  (or  out)  the  next  inorement  of  funds. 

PAT  takes  a  series  of  inputs  and  generates  outputs  in  the  form  of  portfolio-style 
tables  and  various  oharts  and  graphios.  That  is,  viewed  as  a  “blaok  box,”  it  primarily  generates 
displays  to  desoribe  implioations  of  input  information  in  a  struetured  way  (Davis  &  Dreyer, 
2009).  Many  of  the  inputs,  suoh  as  the  investment  options  to  be  oompared,  are  what  one  might 
expeot.  This  oould  inolude,  e.g.,  separate  expenditures  in  the  budget  oategories  of  researoh  and 
development  (R&D),  aoquisition,  and  operations  and  support  (O&S).  Investment  options  may 
differ  in  what  is  to  be  developed  and  how  fast,  in  what  will  be  deployed  operationally,  and  so  on. 
Or  they  may  differ  beeause  of  alternative  teehnical  approaehes  or  beeause  of  alternative 
strategies  (Davis  &  Dreyer,  2009).  PAT’s  outputs  include  color-coded  scoreboards,  which 
compare  options  by  different  objectives  or  measures,  with  red  indicating  poor  and  green 
indicating  good;  tabular  outputs  on  overall  effectiveness  and  cost;  and  standard  charts,  such  as 
charts  of  cost  versus  time  (Davis  &  Dreyer,  2009). 

2.4.2.4.  The  Mitre  Corporation  Portfolio  Analysis  Machine  Tool  (PALMA) 

The  Portfolio  Analysis  Machine  Tool  (PALMA®),  from  the  Mitre  Corporation,  is 
“a  computer  program  that  helps  decision-makers  to  select  the  best  portfolio  (combination)  of 
investments  from  a  set  of  potential  investment  options.”  The  tool  is  based  on  a  mission  tree 
structure,  wherein  a  “strategy-to-task”  tree  is  defined  at  several  levels.  Missions  are  decomposed 
into  sub-missions,  then  further  decomposed  into  subordinate  missions,  functions  and  tasks.  At 
the  lowest  level  (referred  to  as  a  “leaf  node”)  data  for  the  forces  and  systems  that  provide 
relevant  capabilities  are  defined  in  terms  of  effectiveness  and  cost.  PALMA  provides  an 
assessment  of  which  investment  options  should  be  funded  to  achieve  the  desired  goals  for  the 
task.  Like  the  RAND  PAT,  the  displays  are  in  terms  of  stoplight  charts.  The  performance  at  each 
node  in  the  hierarchy  is  specified  by  a  number  from  1  to  100,  representing  value  judgments  by 
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subject-matter  experts  made  at  the  leaf  nodes.  Performance  at  higher  levels  are  computed  as 
weighted  averages  of  the  scores  at  the  contributing  lower  levels  based  on  heuristic  “importance” 
percentage  weights  (again  specified  by  subject-matter  experts).  The  numerical  ranges  are 
translated  into  red-yellow-green-blue  color  codes  for  display  purposes.  By  changing  the  mix  of 
investment  options,  the  performance  at  the  leaf  nodes  will  change,  and  those  changes  will 
“bubble  up”  to  changes  at  the  mid  and  top  levels.  Thus  the  user  can  determine  a  mix  of 
investment  options  that  provide  a  satisfactory  assessment  at  the  mission  level.  To  assist  the  user, 
the  tool  can  display  efficient  frontier  charts  of  cost  versus  effectiveness  to  highlight  which 
investment  choices  provide  the  most  effectiveness  benefit  for  a  particular  investment  level.  The 
least  cost-effective  systems  will  lie  far  away  from  the  efficient  frontier,  which  facilitates  their 
identification  for  tradeoff  analysis.  The  displays  facilitate  exploratory  analysis  by  allowing  the 
user  to  include  or  exclude  an  investment  option  with  a  simple  mouse  click.  A  special  version  of 
the  model  is  also  available  allowing  time-phasing  of  investment  options  and  budgets  across 
years.  PALMA  is  similar  to  the  RAND  methodology,  but  there  are  differences.  For  example, 
PAT  does  not  compute  a  weighted  score  for  the  top-level  assessment  from  sub-tier  assessment. 
Instead,  it  simply  assigns  the  lowest  score  from  the  sub-tier  to  the  next  level.  So  if  four  sub-tiers 
are  rated  “green,  green,  yellow,  red,”  the  next  higher  tier  would  be  rated  “red”  (Porter,  Bracken, 
Mandelbaum,  &  Kneece,  2008). 

2. 4. 2. 5.  Mitre ’s  Matrix  Mapping  Tool  (MMT) 

The  Matrix  Mapping  Tool  (MMT)  was  developed  for  the  Office  of  the  Under 
Secretary  for  Acquisition,  Technology,  and  Logistics,  (OUSD  AT&L)  and  Joint  Staff/J-8  to 
facilitate  cross  organization  coordination  in  support  of  capabilities  based  planning,  analysis,  and 
acquisition.  The  MMT  is  a  database  with  supporting  software  that  documents  relationships 
between  warfighting  activities,  the  UJTL,  systems,  ACTDs,  roadmaps,  and  capability  areas.  It 
allows  for  a  common  set  of  reusable  data  to  support  portfolio  management  (functional, 
operational),  analysis  of  capability  gaps,  and  other  studies  where  it  is  necessary  to  understand  the 
relationships  across  the  dimensions  listed  above.  This  tool  should  be  of  particular  value  to  EoA 
study  groups  because  of  its  ability  to  display  a  wealth  of  the  pertinent  information  about 
acquisition  programs  and  how  they  link  to  capability  areas.  (Porter  et  ah,  2008). 
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SECTION  3. 


IBCT  CPR  SUMMARY 


The  TRADOC  (Training  and  Doctrine  Command)  Analysis  Center’s  (TRAC)  role  in  the 
CPR  process  began  only  recently  when  in  mid-February  2013  the  Director,  TRAC  agreed  to 
provide  analytic  support  to  the  Infantry  Brigade  Combat  Team  (IBCT)  CPR  development  effort. 
The  IBCT  CPR  was  the  first  of  the  so-called  formational  CPRs  (which  we  defined  in  section  2.2) 
in  contrast  to  all  of  the  previous  ones  that  had  been  done  and  classified  as  functional  CPRs  (see 
section  2.2.  for  definition).  The  function-based  CPR  structure  was  put  in  place  several  years  ago 
by  then-VCSA  LTG  Peter  Chiarelli.  Recently,  however,  LTG  Keith  Walker,  the  director  of  the 
Army  Capabilities  Integration  Center  (ARCIC),  directed  a  shift  toward  formation-based  CPRs, 
saying: 


We  don't  fight  as  a  portfolio  —  we  fight  as  formations.  So,  we're  going  to  take  a 
horizontal  look  across  those  portfolios  and  have  a  formation-based  portfolio  review.  We 
have  warfighting  functions  .  .  .  [and]  they  are  not  the  same  as  the  capability  portfolios. 
Then,  in  the  acquisition  world,  we  have  our  [program  executive  officers]  and  they  don't 
exactly  line  up.  Then,  in  the  way  we  resource  our  Army  with  [program  evaluation  groups 
(PEG)],  they  don't  exactly  line  up  either.  There  are  different  bins  and  somehow  we've  got 
to  integrate  them  all.  They  must  inform  our  PEGs  . . .  because  it  provides  better 
information  to  those  folks  who  are  in  the  business  of  resourcing  our  Army  so  they  can 
adjust  (Bertuca,  2013). 

ETG  Walker  also  said  that  he  expected  the  formation-based  CPRs  to  displace  the 
function-based  system  in  the  near  future. 

The  IBCT  CPR  was  centered  on  eliminating  capability  redundancies  in  the 
formation.  The  IBCT  remains  the  Army’s  most  numerous,  versatile,  and  adaptive  combat 
formation.  Its  ability  to  deploy  rapidly  and  operate  in  complex  terrain  distinguishes  it  from  other 
brigade  combat  teams,  such  an  Armor  or  Stryker  BCT,  while  maintaining  its  forcible  entry 
capabilities.  The  inherent  limitations  of  the  IBCT  pose  a  number  of  challenges  for  the  formation. 
In  order  to  enhance  combat  effectiveness,  MCoE  is  focusing  development  of  DOTMLPE 
solutions  for  the  IBCT  on  the  following  priorities:  Operational  Depth  and  Tempo,  Tactical 
Mobility  and  Agility,  Eirepower,  Mission  Command,  and  Sustainment  &  Protection  (House, 
2013).  On  29  July  2013,  MCoE  briefed  the  results  of  the  first  formation-based  CPR,  to  the  Vice 
Chief  of  Staff  of  the  Army  (VCSA).  The  purpose  of  this  briefing  was  to  provide  an  assessment 
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across  Doctrine,  Organization,  Training,  Materiel,  Leader  Development,  Personnel  and  Faeilities 
(DOTMLPF)  domains  of  IBCT  capabilities  and  make  reeommendations  for  functional  portfolio 
prioritization.  With  strong  support  from  other  CoEs,  TRAC,  ARCIC  and  HQDA,  MCoE  assessed 
eurrent  and  projected  strengths  and  weaknesses  in  the  IBCT  formation  aeross  all  warfighting 
functions  and  identified  opportunities  to  expand  combat  effectiveness  through  integrated 
DOTMEPE  solutions  (Flouse,  2013). 

TRAC  became  involved  in  this  effort  rather  “late  in  the  game”,  with  the  Capability  Needs 
Analysis  (CNA)  16-20  already  seheduled  to  begin  in  April  of  2013.  As  a  brief  and  relevant  aside, 
the  CNA  process  is  a  TRADOC-led  assessment  of  the  Army's  ability  to  perform  future 
organizational  and  functional  missions  as  defined  by  joint  and  Army  concepts,  taking  into 
account  existing  and  programmed  DOTMLPF  solutions  (ARCIC,  2013).  TRAC  did  participate 
in  the  event  with  the  intent  of  gaining  insight  into  the  CNA  process  and  acquiring  data  to  support 
IBCT  CPR  solution’s  trade’s  analysis.  TRAC  also  agreed  to  provide  feedbaek  to  ARCIC  with 
reeommendations  for  improving  the  CNA  process.  TRAC  then  leveraged  the  data  eaptured 
during  CNA  16-20,  which  in  reality  was  an  exercise  in  SME  elieitation,  and  developed  a  list  of 
potential  trades  that  the  TRADOC  Capabilities  Manager  (TCM)  IBCT  might  bring  forward  to 
senior  Army  leadership  using  a  goal  program  that  sought  to  maximize  utility  according  to  a  set  of 
prioritized  task-scenario  pairings  subject  to  a  series  of  postulated  funding  level  constraints.  The 
final  brief  that  TRAC-WSMR  provided  can  be  found  at  this  link: 

https://hq.tradoe.armv.mil/sites/trae/Proieets/060097/Shared%20Doeuments/TRAC%20Analvsis 

%20Support%20to%20IBCT%20CPR%20Final%20Brief.pptx 

TRAC-WSMR  has  eontinued  working  with  MCoE  and  the  formational  CPR  proeess  this 
EY  with  developing  another  optimization  program  in  support  of  the  Armor  Brigade  Combat 
Team  CPR,  due  to  be  completed  late  in  the  summer  of  2014.  The  TRAC-MTRY  project  team  not 
only  shared  our  current  work  on  CPR  with  TRAC-WSMR  in  order  to  possibly  give  them  some 
ideas  for  future  direction,  but  also  provided  an  informal  type  peer  review  of  their  optimization 
model  at  several  different  times  during  its  development,  and  will  eontinue  to  do  so  until  the  work 
is  complete. 


18 


SECTION  4. 


ALTERNATE  METHODOLOGY 


This  section  describes  the  main  focus  of  this  research  project,  the  development  of  an 
alternative  trade  space  analysis  to  support  the  CPR  process.  Our  methodology  can  be  divided 
into  five  phases  as  seen  in  Figure  2. 
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Figure  2:  Trade  Space  Analysis  Methodology 

During  the  Subject  Matter  Expert  (SME)  elicitation  process,  we  gather  data  from  the  SMEs,  by 

warfighting  function,  as  to  the  effectiveness  of  a  solution  in  providing  a  pre-defined  required 

capability  through  the  lens  of  a  specific  scenario,  as  well  as  the  overall  risk  of  a  solution  to  said 

required  capability  if  it  is  not  implemented.  During  this  initial  phase,  SMEs  will  also  weight  the 

importance  of  each  required  capability  /  scenario  pairing  using  a  measure  of  consensus.  We  then 

calculate  the  effectiveness  of  a  solution  using  the  SME  effectiveness  scoring  data,  Bayes  Law, 

and  the  weights  from  phase  1  in  constructing  an  additive  value  model.  Next  we  determine  the 

risk  while  using  SME  risk  scoring  data  to  construct  a  distribution  of  risk  scores  across  SMEs  and 
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then  find  the  mean  of  the  super-quantile.  Finally  we  display  the  results  in  terms  of  the  3  key 
elements  that  would  enable  deeision  making;  eost,  effectiveness,  and  risk. 

4,1.  SME  ELICITATION 

The  SME  elicitation  process  that  the  project  team  designed  for  this  effort  is  similar  too, 
though  not  identical,  to  the  current  method  used  by  the  Army,  the  Capability  Needs  Analysis 
(CNA),  and  can  use  some  of  the  data  already  produced  by  the  CNA.  As  such,  it  would  be 
possible  to  only  modify  the  CNA  process  in  order  to  implement  our  alternative  methodology, 
instead  of  completely  supplanting  the  CNA  with  the  “new”  process  presented  here.  In  today’s 
fiscally  constrained  environment,  this  would  be  both  more  feasible  and  practical.  The  elicitation 
process  can  be  divided  into  two  distinct  parts;  the  determination  of  weights,  and  the 
characterization  of  effectiveness  and  risk.  Each  part  will  be  discussed  below  in  turn. 

4.1.1,  Weight  Determination 

The  process  begins  by  dividing  the  SMEs  into  their  respective  “area  of  expertise”  or 
warfighting  function  (WfF),  which  includes  Mission  Command,  Movement  and  Maneuver, 
Intelligence,  Eires,  Sustainment,  and  Protection.  Once  this  is  done,  each  group  of  warfighting 
function  SMEs  will  weigh  the  importance  of  each  required  capability  /  scenario  pairing  using  a 
predetermined  categorical  scale  (i.e.,  a  Eikert  scale).  The  output  of  this  process  will  be  a  single 
weight  for  each  required  capability  /  scenario  pair,  per  WfF,  utilizing  a  measure  of  consensus  to 
combine  all  of  the  disparate  WfF  SMEs  weights  into  a  single  one.  This  measure  of  consensus, 
whose  form  was  inspired  by  the  Shannon  entropy,  is  simply  a  measure  of  the  degree  of  attraction 
to  a  mean  value.  It  is  a  measure  that  characterizes  the  entire  set  of  SMEs  or  stakeholders,  and 
thus  a  measure  of  the  collective  “feeling”  (Tastle,  Abdullat,  &  Wierman,  2010).  Though 
generally  requiring  an  ordinal  scale,  the  consensus  measure  can  be  used  with  interval  and  ratio 
scales,  and  results  in  an  easily  interpretable  number  between  zero  and  one  (unlike  variance  or 
standard  deviation)  (Tastle  et  ah,  2010).  The  equation  for  consensus  is: 

Consensus{X)  =  1-1-  ^  pj  log2  ^  ^  d  ^ 

t=i  '  ^ 
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where  Xrepresents  the  list  of  Likert  categories,  is  an  element  ofX,  fiy-is  the  mean  ofX,  and  d^- 
is  the  width  of  X,  —  ^rnax  “  ^min-  The  mean  of  X  is  given  in  the  usual  way  by  fix  — 

Pi^i-  We  provide  a  small  visual  example  of  what  the  SMEs  would  be  asked  to  inform 
below  in  Figure  3  with  3  required  capabilities  and  only  2  scenarios  to  help  illustrate  this  idea. 


Protection 

Required  Capability 

Scenario 

Weight 

1.1 

EPP 

0.2 

1.1 

NEA 

0.3 

1.2 

EPP 

0.1 

1.2 

NEA 

0.1 

1.3 

EPP 

0.2 

1.3 

NEA 

0.1 

Figure  3:  Example  of  SME  Weight  Elicitation  for  “Protection”  WfF 

An  additional  requirement  is  that  all  of  the  weights,  by  WfF,  must  sum  to  one. 

4,1,2,  Characterization  of  Effectiveness  and  Risk 

Once  the  weights  are  determined,  we  can  move  on  to  characterizing  the  effectiveness  and 
risk.  First,  we  ask  each  of  the  SMEs  for  each  solution  and  required  capability  /  scenario  pairing, 
by  WfF,  to  first  determine  the  effectiveness  of  a  solution  by  answering  the  following  question; 
“How  effective  is  this  particular  solution  in  providing  this  required  capability,  given  this  (a) 
particular  scenario?”  The  answer  is  a  categorical  in  nature,  using  a  Likert  scale  type  response. 
The  actual  number  of  scale  levels  as  well  as  the  exact  wording  of  the  responses  is  left  up  to  a 
study  lead  and  not  critical  to  this  effort. 

For  the  risk  determination  of  each  solution  (by  required  capability  /  scenario  pairing,  by 
WfF),  we  follow  the  well-known  and  accepted  methodology  laid  out  in  the  Risk  Assessment 
Matrix,  found  in  FM  5-19.  We  ask  the  SMEs  to  determine  the  two  parts  of  risk,  probability, 
“What  is  the  likelihood  of  the  scenario  under  consideration,”  and  severity,  “What  is  the 
consequence  to  providing  this  required  capability  if  this  solution  is  not  implemented.”  These  are 
then  combined  into  an  overall  risk  category  as  seen  in  the  matrix  below  where  4  -  Extremely 
High,  3  -  High,  2  -  Moderate,  and  1  -  Low. 
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Probability 

Severity 

Frequent 

Likely 

Occasional 

Seldon 

Unlikely 

Catastrophic 

4 

4 

3 

3 

2 

Critical 

4 

3 

3 

2 

1 

Marginal 

3 

2 

2 

1 

1 

Negligible 

2 

1 

1 

1 

1 

Table  1:  Risk  Classifications 


As  in  section  4.1.1.,  we  provide  a  small  visual  example  below  in  Figure  4  using  12  SMEs  (2  per 
WfF),  3  solutions,  3  required  capabilities,  and  2  scenarios. 


Figure  4:  SME  Elicitation  Effectiveness  and  Risk  Example  Table 

These  effectiveness  and  risk  values  will  be  used  in  subsequent  parts  of  this  methodology. 

4,2.  CALCULATING  EFFECTIVENESS  USING  BAYES  LAW 

We  begin  with  a  discussion  on  the  theoretical  underpinnings  of  using  Bayes  Law  in 
conjunction  with  the  SME  data  in  order  to  calculate  how  effective  a  solution  is  in  providing  a 
specific  required  capability  given  a  scenario,  and  then  provide  an  example  with  actual  SME 
elicited  data  for  a  more  complete  understanding  of  the  process. 
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4,2.1,  Theoretical  Application  of  Bayes  Law 

Suppose  a  survey  is  condueted  during  whieh  S  subjeet  matter  experts  (SMEs)  assess  the 
effeetiveness  of  solutions  to  a  required  eapability  and  scenario  pair  with  a  score  which  is  an 
element  of  the  set  of  ordered  categories!  1,  1  is  the  lowest  category  and  C  is  the  highest. 

We  will  subsequently  refer  to  the  required  capability  and  scenario  pair  as  simply  a  double  when 
no  confusion  will  result.  We  make  the  following  initial  assumptions: 

1 .  All  SMEs  are  statistically  the  same;  that  is,  they  are  randomly  drawn  from  the  same 
population. 

2.  The  SMEs  score  a  solution  to  a  double  independently  of  the  other  SMEs,  the  other 
solutions,  and  the  other  doubles. 

3.  Each  double,  t,  has  an  unobservable  random  probability  that  a  SME  will  assess 

score  c  to  solution  s.  The  conditional  distribution  of  the  number  of  S  SMEs  that  assess  score  c  to 
solution  5  for  the  double  t  given  Pt(s)  =  —  >  Pt,c(.^))  is  multinomial  with  S  trials  and 

probabilitiesPt(s)  =  ,  Pt,c(s)). 

The  distribution  of  Pt(s)  is  called  the  Bayesian  prior  distribution.  The  conditional 
distribution  of  Pt(s)  given  the  result  of  the  survey  is  called  the  Bayesian  posterior  distribution. 
The  posterior  distribution  allows  the  evaluation  of  the  conditional  probability,  given  the  results 
of  the  survey,  of  events  such  as  the  event  that  more  than  (p  x  100)%  of  future  SMEs  will  assess 
at  least  score  c  to  solution  s  for  double  t  for  0  <  />  <  1 . 

The  posterior  distribution  depends  upon  the  survey  results  and  the  prior  distribution.  If 
the  prior  distribution  is  Dirichlet,  then  the  posterior  distribution  is  also  Dirichlet  with  parameters 
determined  by  the  survey  results  and  the  parameters  of  the  prior  distribution  (we  use  Dirichlet 
since  it  is  the  conjugate  prior  to  the  Multinomial  distribution  and  will  be  discussed  in  more  detail 
later  on  in  the  report)  (DeGroot,  1970).  When  there  is  no  a-priori  information,  a  common 
Dirichlet  prior  is  to  set  all  the  shape  parameters  equal  to  h  >  0.  The  closer  b  is  to  0,  the  greater 
the  influence  of  the  survey  results  on  the  posterior  distribution. 
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The  Dirichlet  distribution  is  a  generalization  of  the  beta  distribution.  Letting  T  denote  the 
gamma  function,  a  random  variable  P  =  (Pi, ,  P^)  has  a  Dirichlet  distribution  with  shape 
parameters  (ai, ... ,  a^)  >  0  if  its  density  function  is  of  the  form, 


/(pi,...,Pc) 


r(ai  +  a2  +  •■•  +  az-l  ac-l 

r(ai)r(a2)x...xr(a.)^‘ 


for  0  <  Pc  <  1 


with  (pi, ...  Pc)  such  that  2]c=i  Pc  —  0  otherwise. 


A  sequential  procedure  to  allocate  resources  to  solutions  uses  the  following  criteria  for 
solutions  and  doubles:  an  assessment  of  the  likelihood  of  the  double  occurring;  a  minimum 
acceptable  category  for  the  solution,  Cmin  (e.g.  solution  must  be  assessed  at  least  category  c);  a 
minimum  acceptable  percentage  of  future  SMEs  that  will  assess  that  category  or  higher  to  the 
solution,  (a  measure  of  consensus  among  the  SMEs);  a  minimum  posterior  probability  that  more 
than  that  percentage  of  SMEs  will  assess  the  minimum  acceptable  score  or  higher  to  the  solution, 
(1  minus  this  posterior  probability  is  the  risk  that  a  smaller  percentage  of  future  SMEs  will  assign 
category  Cmin  to  the  solution). 

IfP  =  (Pi,  ...,Pc)  has  a  Dirichlet  distribution  with  shape  parameters  (cTi, ... ,  a^)  >  0, 
then  for  c  G  (1,2, ... ,  C],  Y.i=c  has  a  beta  distribution  with  parameters  (l]f=c  ‘^t)- 

particular,  the  marginal  distribution  of  Pc  is  beta  with  parameters,  (a^,  'Zi=^c  '^t)-  Thus,  [Pc]  = 

etc 

2f=i  cci  ' 


We  will  combine  the  Dirichlet  with  the  Multinomial  models  for  this  effort.  Assume  that  a 
SME  will  assess  an  effectiveness  score  c  to  a  solution  to  one  required  capability  scenario  pair 
with  unobservable  probability  independently  of  the  other  SMEs.  Also  assume  P  = 

(Pi . Pc)  has  a  Dirichlet  distribution  with  shape  parameters  a  =  (a^, ... ,  a^);  this  distribution 

is  called  the  Bayesian  prior  distribution.  Eet  be  the  number  of  the  S  SMEs  that  assign 
effectiveness  score  c  to  the  solution.  Put  iV  =  (iVi, ... ,  N^).  Assume  the  conditional  distribution 
of  iV  given  P  =  (Pi, ... ,  Pc)  is  multinomial.  Note  that  the  distribution  of  P  in  general  depends  on 
the  required  capability  and  scenario. 

The  Dirichlet  distribution  is  the  conjugate  prior  for  the  multinomial  distribution.  The 
conditional  distribution  of  P  =  (Pi, ... ,  Pc)  given  the  results  of  the  survey,  iV  =  (iVi, ... ,  N^)  = 
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(n^, ... ,  Tie)  ,  is  Dirichlet  with  shape  parameters  +  n^, ... ,  a^.  +  n^);  this  conditional 
distribution  is  called  the  Bayesian  posterior  distribution.  Note  that 


E[Pc\Ni  =  ni,  ...,Nc  =  nj  = 


Ur  +  Ur 


ar 

^  1 

Uc 

Thus,  the  posterior  expected  value  of  Pc  given  the  survey  results  is  a  weighted  sum  of  the 
prior  expected  value  of  Pc  and  the  maximum  likelihood  estimator  which  is  the  fraction  of  the  S 
SMEs  that  assign  score  c  to  the  solution. 

When  there  is  no  a-priori  information,  a  common  Dirichlet  prior  is  to  set  CTj  =  ^  D  = 

1, ... ,  C  for  some  constant  h  >  0.  In  this  case. 


E[PcWi  =  n^,...,Nc  =  Tie] 


b 

1 

Uc 

b  +  Vi=i  rci 

b  +  Ui 

The  constant  b  determines  sensitivity  of  the  posterior  distribution  to  the  survey  data,  and  is 
sometimes  referred  to  as  the  prior  strength. 


4,2,2,  Example  Application 

We  next  provide  an  example  of  implementing  Bayes  as  described  above  with  some 
survey  data  captured  by  the  project  team  at  Ft.  Lee  during  a  Tactical  Wheeled  Vehicle  workshop. 
Though  this  SME  elicitation  workshop  was  not  set-up  and  executed  as  we  describe  our  process 
in  section  4.1,  it  will  still  provide  the  reader  with  an  understanding  of  how  we  would  apply 
Bayes. 

Subject  matter  experts,  (SMEs),  participate  in  a  number  of  surveys  to  consider  possible 
solutions  to  20  required  capabilities.  The  same  9  possible  solutions  are  considered  for  all  the 
required  capabilities.  The  number  of  SMEs  that  participate  in  the  surveys  is  not  constant. 

A  SME  assigns  a  score  of  1,  2,  3,  or  4  to  each  solution  for  a  required  capability.  The 
scores  are:  l=low  effectiveness;  2=medium  effectiveness;  3=high  effectiveness;  and  4=very  high 
effectiveness. 
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Let  Pf-  c(.s)hQ  the  (unobservable)  probability  a  SME  assigns  seore  c  to  solution  s  for 
required  capability  t.  Assume  Pt(5)  =  has  a  Dirichlet 

Bayesian  prior  distribution  with  all  shape  parameters  equal  to  0.25.  We  initially  assume  that 
SMEs  score  the  solutions  for  each  required  capability  independently  of  each  other  and  the  scores 
given  to  other  required  capabilities  and  solutions.  The  Bayes  posterior  distribution  of  the 
probabilities  a  SME  assigns  score  c  e  (1,2, 3, 4}  to  solution  5  for  required  capability  t, 
also  has  a  Dirichlet  distribution. 

What  is  unique  about  this  strategy  is  that  is  allows  us  to  make  statistical  inferences  about 
the  entire  SME  population,  not  simply  from  the  sample  population.  Additionally,  this  approach 
gives  us  the  flexibility  to  eliminate  or  trim  solutions  that  don’t  meet  a  minimum  criterion  prior  to 
using  the  optimization  model,  thereby  reducing  the  amount  of  input  data  to  the  model.  Eor 
instance,  we  can  choose  the  following  criteria  for  a  solution,  required  capability/scenario  pair; 

•  Choose  a  minimum  acceptable  SME  score  category,  Cmin  e.g.  3=high 
effectiveness. 

•  Choose  a  minimum  probability,  0  <p  <\,  that  more  than  {p  x  100)%  of  new 
SMEs  from  the  same  population  of  SMEs  will  assign  a  score  of  or  greater  to  a 
solution. 

•  Choose  a  minimum  posterior  probability  that  at  least  (p  x  100)%  of  new  SMEs 
will  assign  a  score  of  Cmin  or  greater. 

Those  solutions  with  posterior  probability  greater  than  or  equal  to  the  minimum  posterior 
probability  will  be  considered  for  the  required  capability/scenario  pair. 

We  can  perhaps  illustrate  this  better  in  tabular  format.  Table  1  displays  the  maximum 
posterior  probability  over  all  solutions  for  each  required  capability  t  that  more  than  70%  of  new 
SMEs  will  give  a  score  greater  than  or  equal  to  c  for  c  =  2,  3,  4.  The  choice  of  70%  is  arbitrary; 
the  posterior  probabilities  will  decrease  for  80%  and  will  increase  for  60%.  In  the  third  column 
of  Table  1  only  required  capabilities  3,  6,  9,  and  12  have  at  least  one  solution  with  posterior 
probabilities  greater  than  0.9  that  more  than  70%  of  SMES  will  score  the  solution  highly 

effective  or  very  highly  effective,  (score  ^  3).  In  the  second  column  of  Table  1  there  are  no 
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solutions  for  required  eapabilities  1,  4,  and  16  with  posterior  probability  greater  than  0.5  that 
more  than  70%  of  future  SMEs  will  score  the  solution  with  at  least  medium  effectiveness,  (score 

^  2).  In  the  fourth  column  there  is  at  least  one  solution  for  required  capability  12  for  which  the 

posterior  probability  is  0.75  that  more  than  70%  of  future  SMEs  will  score  the  solution  very 
highly  effective,  (score  =  4);  solution  9  has  posterior  probability  0.75  that  more  than  70%  of  new 
SMEs  will  score  it  very  highly  effective  for  required  capability  12. 


The  Maximum  Posterior  Probability  Over  9  Solutions  that  More  than  70%  of 
Euture  SMEs  will  Score  a  Solution  c  or  Higher  for  c=2,  3,  4 

Required 

Capability 

Maximum  Posterior 
Probability  Over  9 
Solutions  More  than  70% 
of  Future  SMEs  will 
Score  the  Solution  >=  2 

Maximum  Posterior 
Probability  Over  9 
Solutions  More  than  70% 

of  Future  SMEs  will  Score 

a  Solution  >=  3 

Maximum  Posterior 
Probability  Over  9 
Solutions  More  than 

70%  of  Future  SMEs  will 
Score  a  Solution  =  4 

1 

0.28 

0.00 

0.00 

2 

0.87 

0.00 

0.00 

3 

0.99 

0.98 

0.43 

4 

0.06 

0.00 

0.00 

5 

0.98 

0.00 

0.00 

6 

1.00 

1.00 

0.15 

7 

0.55 

0.00 

0.00 

8 

1.00 

0.00 

0.00 

9 

1.00 

0.91 

0.00 

10 

1.00 

0.00 

0.00 

11 

1.00 

0.00 

0.00 

12 

1.00 

1.00 

0.75 

13 

0.52 

0.00 

0.00 

14 

1.00 

0.00 

0.00 

15 

1.00 

0.98 

0.01 

16 

0.35 

0.00 

0.00 

17 

0.97 

0.00 

0.00 

18 

1.00 

0.99 

0.26 

19 

1.00 

1.00 

0.00 

20 

1.00 

0.00 

0.00 

Table  2:  SME  Elicitation  Probabilities 


This  example  illustrates  how  we  can  thus  eliminate  solutions,  by  warfighting  function,  which 

does  not  meet  the  minimum  criteria  that  we  choose. 
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4,2,3,  Effectiveness  Scoring 

Once  we  have  determined  the  probabilities  associated  with  each  solution,  by  required 
capability  /  scenario  pair,  by  WfF,  we  will  use  an  additive  value  model  using  the  weights  found 
during  the  SME  elicitation  process  and  the  probabilities  that  we  found  in  the  previous  section 
using  Bayes  Law. 

First  we  define  the  following: 

i  =  number  of  required  capabilities, 
j  =  number  of  scenarios, 
k  =  number  of  solutions. 


/=  number  of  warfighting  functions. 

So,  for  each  of  the  WfF’s,  by  solution,  we  calculate  the  effectiveness  as: 

i  j 


11  weightfij  x  probabilityfijif^ , 


1  1 


In  words,  this  means  that  we  multiply  each  of  the  required  capability  /  scenario  weights  by  the 
corresponding  probability  and  then  sum  over  each  solution,  by  WfF.  Or,  using  our  small  visual 
example  with  only  3  solutions  we  see: 


Mission  Command 


So 


Movement  &  Maneuver 


So 


Intellige  nee 


So 


Fires 


So 


Sustainment 


So 


Protection 

Solution 

Effective  ness 

SI 

0.94 

S2 

0.88 

S3 

0.S4 

Figure  5:  Example  Effectiveness  Calculation 

And  then  for  each  solution,  across  all  WfF,  the  final  effectiveness  score  is  given  by: 


6 

^  EffectivenesSfi^,  V^. 
/=i 
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Once  again,  in  words,  this  means  that  we  sum  the  effeetiveness  values  over  all  of  the  WfF,  by 
solution,  in  order  to  obtain  a  single  effeetiveness  seore  for  eaeh  of  the  solutions.  This 
effectiveness  seore  will  be  what  we  are  trying  to  maximize  in  our  optimization  model.  And 
again,  a  small  visual  example  yields: 


Effectiveness ScoresAcross all  WfF 

WfF 

SI 

S2 

S3 

Mission  Command 

0.94 

0.88 

0.00 

Movements  Maneuver 

0.45 

0.23 

0.71 

Intelligence 

0.97 

0.36 

0.29 

Fires 

0.42 

0.55 

0.11 

Sustainment 

1.03 

0.63 

1.05 

Protection 

0.86 

1.13 

0.91 

Total 

4.67 

3.78 

3.07 

Figure  6:  Effectiveness  Scoring  Example 
4.3.  CALCULATING  RISK 

Our  methodology  for  calculating  risk  consists  of  two  parts.  First,  we  estimate  a  single 
probability  distribution  for  eaeh  solution  that  gives  the  frequeney  with  whieh  an  arbitrary  SME 
will  respond  with  a  specific  risk  category  (as  described  in  seetion  4.1.2.).  Seeond,  we  use  the 
estimated  density  to  eompute  the  superquantile,  thereby  arriving  at  a  single  risk  seore  per 
solution.  We  briefly  diseuss  each  component  in  turn. 

We  determine  a  density  that  maximizes  the  likelihood  of  receiving  the  SME  risk 
responses  that  were  actually  received  during  the  SME  elicitation  process,  and  build  a  single  and 
distinct  density  for  each  solution.  In  reality,  the  total  number  of  densities  per  solution  would 
equal:  (the  #  of  required  capabilities)  x  (#  of  scenarios)  x  (#  of  WfF).  We  would  aggregate  all  of 
these  into  one  density  per  solution  by  simply  combining  all  of  the  risk  responses  and  eorrectly 
elassifying  their  frequeneies.  The  wavy  line  in  Figure  7  is  a  notional  estimate  based  on  the  given 
responses  and  the  information  that  such  a  density  should  be  unimodal. 
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0  Low  Mod  High  Ext.  high 

Figure  7:  Example  Risk  Density 

The  superquantile  is  a  particularly  useful  way  of  numerically  quantifying  the  inherent  risk 
and  variability  of  a  particular  distribution,  in  this  case  the  risk  density.  A  superquantile  is  the 
conditional  average  of  a  fraction  of  the  most  pessimistic  SMEs.  This  risk  superquantile  will  then 
serve  as  a  constraint  in  our  optimization  model. 

For  example,  Figure  8  illustrates  a  superquantile  at  the  80%  level:  look  at  the  100  -  80  = 
20%  most  pessimistic  SMEs  according  to  the  density  in  the  shaded  area.  This  is  a  number  that 
then  can  be  communicated  to  a  decision  maker  as  a  reasonable  conservative  estimate,  but  not 
overly  so.  It  is  a  number  that  takes  into  account  variability  in  the  SME  responses  as  well  as  the 
importance  of  looking  at  asymmetric  variability.  We  might  not  be  as  interested  in  variability  on 
the  low  (optimistic)  side  as  on  the  high  (pessimistic)  side.  Consequently,  the  use  of  the  standard 
deviation  would  not  be  appropriate. 


Figure  8:  Example  Superquantile 

4.4.  CAPITAL  BUDGET  OPTIMIZATION  MODEL 

The  mathematical  model  used  for  formulation  is  the  basic  capital  planning  structure 
(Brown,  Dell,  &  Newman,  2004).  In  general,  the  solutions  enter  the  optimal  portfolio  through 
maximizing  their  effect  while  meeting  the  risk  demand  and  not  exceeding  the  supply  of  capital. 
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As  it  is  currently  constructed,  the  model  is  for  a  single  time  period  with  the  assumption  of 
independent  and  mutually  exclusive  proposed  solutions. 

The  model  can  be  extended  to  consider  variable  and  fixed  costs  through  the  introduction 
of  additional  budget  constraints.  Additionally,  multi-time  periods  can  be  introduced  through 
indexing.  The  multistage  model  would  require  additional  constraints  but  would  no  longer  need 
the  assumption  of  independence  or  mutually  exclusive  solutions.  The  decisions  to  include  these 
extensions  would  be  based  primarily  on  data  availability. 


4,4.1.  Mathematical  Formulation 
Sets 

k  G  /f :  set  of  all  solutions  possible  for  portfolio  investment. 
Parameters 

=  cost  per  solution  k  [$] 

b  =  acceptable  budget  for  portfolio  investment  [$] 
r^.  =  risk  associated  with  solution  k 
d  =  minimum  acceptable  risk 
Yk  =  effectiveness  of  solution  k 


Binary  Variable 


_  ri,  if  solution  k  is  purchased^ 
1  0,  otherwise  j 


Problem 


Maximize  effectiveness:  The  objective  is  to  maximize  effectiveness  associated  with 
optimal  portfolio  selection.  The  objective  function  includes  the  effectiveness  coefficient  (Score) 
calculated  using  SME  input  and  Bayes  Law  for  each  solution. 


max 


kEK 


Subject  to: 
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Acceptable  risk:  This  constraint  dictates  the  minimum  acceptable  risk.  Solutions 
must  achieve  a  minimum  cumulative  risk  score  ensuring  the  optimal  portfolio  meets  a 
determined  risk  level. 

kEK 

Budget  Limitations:  Limits  the  amount  of  cost  to  the  budgeted  amount. 

^  CkYk  <  b 

kEK 

Integrality:  Ensures  all  of  the  variables  are  restricted  to  be  binary. 

inary 

The  output  for  this  optimization  model  is  a  portfolio  of  solutions  with  a  maximized 
effectiveness  that  achieves  both  a  minimum  level  of  risk  as  well  as  cost.  The  model  can  be  run 
by  maximizing  the  effectiveness  of  each  solution  individually,  or  grouping  portfolio  solutions 
into  different  Courses  of  Action  (COAs)  that  would  change  by  varying  the  acceptable  risk  score, 
budget  considerations,  or  the  number  and  identity  of  fixed  solutions. 

4,5.  VISUALIZATION 

In  designing  a  visualization  strategy,  the  project  team  fist  had  to  discern  what  the  most 
important  elements  are  that  a  decision  maker  must  consider  when  evaluating  the  feasibility  and 
desirability  of  a  particular  portfolio.  For  this  effort,  we  identified  effectiveness  (which  we  again 
define  as  “How  effective  is  a  particular  solution  in  providing  a  required  capability,  given  a 
scenario?”),  risk  (based  on  probability,  severity,  and  the  superquantile  of  the  overall  risk 
distribution  by  solution),  and  cost.  Figure  9  shows  an  example  of  visualizing  the  three  key 
elements  in  two  dimensions  with  the  effectiveness  score  on  the  y-axis,  risk  on  the  x-axis,  and  cost 
portrayed  as  the  relative  diameter  of  the  circle;  the  larger  the  circle,  the  more  expensive  the 
solution  or  COA.  As  Figure  9  is  currently  shown,  we  have  only  “Fow”  and  “High”  labels  on  the 
axes.  This  could  easily  be  modified  to  show  actual  risk  and  effectiveness  scores,  or  left  as  is  in 
order  to  be  more  general  for  ease  of  use. 
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Effectiveness  vs.  Risk 

Cost  ■  Circle  Diameter 


Figure  9:  Visualization  of  the  3  Key  elements 


Ideally,  we  would  like  to  see  a  solution  or  COA  in  the  top  left  portion  of  the  chart  with  a  small 
diameter  circle  (the  highlighted  green  portion),  meaning  that  it  is  low  risk,  highly  effective,  and 
low  cost  solution.  Conversely,  using  this  visualization  tool  highlights  the  less  desirable  solutions, 
and  allows  a  decision  maker  to  eliminate  those  that  meet  specified  criteria. 
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SECTION  5. 


SUMMARY 


5.1.  WAY  AHEAD 

The  difficulty  with  this  approach  is  in  calculating  the  posterior  distribution,  i.e.,  the 
probabilities,  while  making  the  assumption,  one  that  is  not  always  true,  that  the  prior  is  Dirichlet 
simply  because  the  SME  solution  scores  have  a  multinomial  distribution.  Additionally,  updating 
the  shape  parameters  for  the  distribution  can  be  time  consuming  and  somewhat  troublesome. 

There  are  numerous  directions  for  follow-on  work  from  this  effort.  Other  statistical 
techniques,  vice  using  Bayes,  could  be  implemented  in  order  to  calculate  an  effectiveness  score 
from  SME  input.  While  this  effort  did  not  explore  additional  alternate  methodologies,  certainly 
one  could  appreciate  that  any  non-parametric  or  distribution  free  techniques  that  make  no 
assumptions  about  the  distribution  of  the  SME  solution  scores  would  be  useful  in  finding  the 
overall  effectiveness  of  a  solution. 

5.2.  CONCLUSIONS  AND  RECOMMENDATIONS 

At  its  core,  this  effort  has  sought  to  develop  an  alternative  trade  space  analysis  technique 
for  use  in  future  CPR  work  that  TRAC  may  take  part  in.  By  using  Bayes  to  calculate 
probabilities  that  future  SMEs  will  assign  certain  effectiveness  scores  to  certain  solutions  (given 
a  required  capability  /  scenario  pair)  and  then  weighting  these  scores  according  to  importance 
(also  derived  from  SMEs),  we  attempt  to  capture  how  well  a  solution  provides  a  required 
capability  in  a  given  situation.  We  also  endeavor  to  capture  the  inherent  risk  in  a  solution  of  not 
meeting  a  capability.  We  take  these  two  factors,  combined  with  available  cost  and  budget 
information,  and  select  a  portfolio  of  solutions  with  a  maximized  effectiveness  that  achieves  both 
a  minimum  risk  and  budget.  Eastly,  we  visualize  these  three  key  elements  on  a  single  graph  in  a 
way  that  makes  it  obvious  which  solutions  are  the  “best”,  meaning  those  that  have  the  lowest 
cost  and  risk  but  the  highest  effectiveness. 
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